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Importance 


A  Generic  Example  -  Comparing  Four  Defects 


o 


Which  would  you 
fix  first? 


>■ 
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How  do  we  judge  importance? 


Using  “severity”  alone  has  issues 

•  People  are  tempted  to  negotiate  a  severity  rating  to  account  for 
the  importance  they  perceive 

•  Without  a  way  to  discuss  what  makes  things  important,  the 
conversation  may  become  a  competition  among  advocates 

RPN  focuses  on  risk  exposure 

•  Allows  the  team  to  assess  the  priority  of  fixes 

•  Can  relate  priority  to  the  understanding  of  risk 

Risk  can  be  perceived  from  different  viewpoints 

•  User,  developer,  cost,  time 

•  May  need  multiple  views  to  make  the  best  decision 
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RPN  General  Explanation  -1 


Generally  based  on  processes  that  were  developed  from 
reliability  and  cost  methods 

•  Severity:  a  rating  of  the  adverse  impact  of  the  defect  - 

a  measure  that  reflects  the  negative  consequence  to  the  users 
or  developers 

•  Occurrence:  how  often  the  defect  is  encountered  and/or  how 
long  it  takes  to  recover  functionality  -  a  measure  that  reflects  a 
different  element  of  the  impact  of  the  defect 

•  Detection:  how  easy  it  is  to  spot  the  defect  is  when  it  occurs  - 
a  measure  that  reflects  the  risk  of  unmitigated  consequences  if 
the  defect  is  not  remedied 
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RPN  General  Explanation  -2 


For  weapon  systems  these  may  equate  to: 

•  Severity  =  Threat  to  mission  success  (Operational  and  System) 

•  Occurrence  =  How  often  it  happens,  how  much  time  to  recover 

•  Detection  =  Ability  to  detect  that  the  problem  has  occurred 
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RPN  General  Explanation  -3 


RPN  includes: 

•  Rating  scales  characterizing  elements  of: 

•  Severity, 

•  Occurrence 

•  Detection 

•  Scaling  values  for  the  ratings 

•  (Optional)  Weighting  for  each  rating  scale  to  emphasize  what 
matters  most/least  in  a  given  system 

RPN  =  Severity  x  Occurrence  x  Detection 

•  A  weighted  sum,  rather  than  multiplying  the  numbers  together, 
can  be  included  an  option 
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Polling  Question 


Would  you  like  us  to  explain  the  basic  premise  of  RPN  in  greater 
detail? 

□  Yes 

□  No 
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Expected  Range  of  Application 


Development,  operation,  and  sustainment  contexts  are  all 
candidates  for  adapting  RPN  to  support  decision  making  on  which 
defects  to  fix  first 

Keys  to  successful  usage 

•  Custom  rating  scales  developed  with  appropriate  personnel 

•  Socializing  draft  materials  with  stakeholders 

•  Buy-in  from  participants  in  existing  defect  review  processes 
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Example  Usage  -  scenario 


A  major  weapon  system  in  early  fielding  is  looking  for  a  way  to  plan 
the  contents  of  releases  comprised  of  DR  fixes 

•  Diverse  user  community  with  legitimate  competing  priorities 

•  Limited  funding  for  future  work  (many  DRs  will  never  be  fixed) 

•  Program  office  motivated  to  maximize  system  utility/value 


Program  Office 


Example  Usage  1 


1 .  A  small  working  group  was  formed 

•  Representatives  familiar  with  existing 
DRs  for  this  system 

•  A  member  of  the  program  office  staff 
who  understands  the  vision  for  the 
system 

•  Measurement  coach  who  can  help 
navigate  the  process  of  constructing 
measurement  scales 


•  Draft  rating  scales  were  developed 
as  well  as  computation  procedures 
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Example  Usage  -  2 


3.  Draft  materials  were  reviewed  with 
user  communities 

•  The  reasons  for  using  RPN  were 
explained  and  tied  to  the  current 
decision  processes 

•  The  rating  scales  were  explained  to 
people  who  write  DRs  or  who 
champion  DRs  to  be  included  in 
releases 


•  Worked  examples  of  real  defects  to 
discuss  how  ratings  are  assigned 

4.  Rating  scales  and  procedures  were 
updated  based  on  feedback 
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Sample  Scales 


The  following  example  covers  scales  developed  to  fit  a  specific 
context,  with  active  involvement  of  stakeholders. 
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Rating  Scales  -  Severity  -  System  Function 


Minor  System  Malfunction 


System  Malfunctions  or  Fails  to  Execute  Some 
Functions  but  work-around  exists 


Interruption  in  System  Functionality  Requiring 
operator  intervention 


Interruption  in  System  Functionality  Requiring 
contractor  Intervention 


Severely  Constrained  System  Functionality — 
difficult  work-arounds  needed 


No  functionality  is  available  and  task  cannot  be 
performed  by  any  method. 


N/A 
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Rating  Scales  -  Severity  -  Operational  Impact 


Increases  operator  workload  slightly 


Increases  operator  workload  significantly 


Could  limit/delay  mission  operations 


Certain  delay/limit  to  mission  operations 


Could  cause  mission  failure 


Certain  mission  failure 


N/A 
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Rating  Scales  -  Detection 


1 

2 

3 

4 

5 


System 


System 


System 


System 


There  is  an  explicit  alert  or  warning  that  there  is  a 
malfunction;  or  the  system  or  application  fails  or  crashes. 


Users  will  always  notice  a  visible  malfunction, 
and  only  novices  would  fail  to  detect  the  unexpected 
system  behavior. 

Users  will  always  notice  a  visible  malfunction,  but  only 
after  other  functions  or  workflow  steps  have  completed. 


A  user  may  detect  subtle  symptoms  during  normal 
operation,  but  may  not  immediately  recognize  the  cause. 


Issue  not  detectable  during  normal  operation 
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Rating  Scales  -  Occurrence 


1  o 

Under  10  hours  to  recover 

2  O0 

Less  than  a  week  to  recover 

3  000 

About  a  week  to  to  recover 

4  <§>00 

Weeks  to  months  to  recover 

5<a©p 

Up  to  3  months  to  recover 

6  Q&Q 

More  than  3  months  to  recover 

Note:  Occurrence  =  Number 
of  times  the  defect  is 
encountered  per  year  x  the 
time  restore  functionality 
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Polling  Question  2 


We  discussed  two  scales  that  equated  to  Severity  -  you  could  use 
additional  scales  for  other  forms  of  severity  and  you  could  also  use 
multiple  scales  for  detection  or  occurrence. 


Would  you  like  to  see  more  examples  of  these  types  of  scales  or 
continue  on  to  how  these  scales  are  used? 


□  More  examples 

□  Continue 
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Using  Proportional  Scales 


RPN  is  based  on  the  use  of  proportional  scales 


The  ordinal  discussed  in  the  last  few  slides  must  be  changed  to  a 
proportional  rating 

Proportional  Ordinal 

■  1  1 


2 

3 

4 

5 

6 
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RPN  -  An  Example  -  Weighted  Average 


Based  on  user  input  the  final  weighed  average  was: 

Scaled  System  Behavior  rating  scale  value  *  10%  + 

Scaled  Operational  Impact  scale  value  *  50%  + 

Scaled  Detection  rating  scale  value  *  20%  + 

Scaled  Time  scale  value  *  20% 

Resulted  in  a  non-continuous  rating  scale  from  0  to  2400 

Note:  The  four  values  could  also  have  just  been  multiplied 
together,  using  different  scales  to  adjust  for  importance 
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Polling  Question 


Would  you  like  us  to  discuss  the  use  of  proportional  scales  and 
ways  to  combine  the  scales  or  continue  with  a  discussion  of  how  to 
use  the  RPN  numbers 


□  More  discussion  of  scales 

□  Continue  with  how  to  use  the  RPN  numbers 
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Resource  Available 


For  a  more  complete  discussion  of  the  examples  presented  here, 
please  download  the  white  paper  available  at  the  following  URL: 

http://resources.sei.cmu.edu/asset  files/whitepaper/2013  019  001  70276.pdf 
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Sample  Data  Description 


For  the  sample  data  we  have: 

Three  users  -  A,  B,  and  C  with  10  DRs  each 

Five  Functions 

•  Communications 

•  Navigation 

•  Planning 

•  Propulsion 

•  Security 

Assume  DRs  will  be  fixed  in  increments  of  3,000  Source  Lines  Of 
Code  (SLOC)  each  (Note:  SLOC  is  used  as  a  proxy  for  cost) 

Even  with  this  small  sample  there  are  hundreds  of  combinations! 
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SLOC 


One  way  to  look  at  the  sample  data 


RPN  Vs.  SLOC 


1400 


1200 


1000 


800 


600 


♦ 


Higher  impact, 
lower  cost  area 


400 


200 


♦  ♦  ♦  / 


♦  ♦ 


500 


1000 


1500 


RPN 


2000 


2500 


Note:  In  this  example,  SLOC  is  being  used  as  a  proxy  for  cost 
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Four  Analysis  Methods 


Method 

Brief  Description 

Pros 

Cons 

Functionality 

Group  DRs  by  system 
function  using  RPN  and 
SLOC  to  select  order 

-  Easier  to  test  specific 
functional  areas 

-  Should  see  improvements 
in  specific  areas  addressed 

-  May  not  address  top  user  ranked  DRs 

-  Some  functional  areas  will  not  be 
addressed  in  every  increment 

-  Some  functional  areas  may  still  need  to 
be  split  due  to  SLOC  constraints 

System  Risk 

List  DRs  by  RPN  and  draw 
a  line  at  the  3000  SLOC; 
Best  used  for  pure 
maintenance  (regression 
testing  only) 

-  Addresses  system  level 
risk  first 

-  Fairly  easy  to  use 

-  Doesn’t  specifically  address 
functionality  groups 

-  Doesn’t  specifically  address  user 
rankings 

User  rankings 

List  DRs  by  user  rankings 
and  draw  a  line  at  3000 
SLOC; 

-  Addresses  user  rankings 

-  Fairly  easy  to  use 

-  May  fix  DRs  with  lower  overall  system 
risk  earlier;  Doesn’t  address  system 
value 

-  Doesn’t  specifically  address 
functionality  groups 

-  Need  to  address  differences  between 

users 

Hybrid 

Combinations  of  the 
methods  above 

Depends  on  method 

Depends  on  method 
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Analysis  Method  -  Functionality 


Look  at  top  level  data  in  a  summary 
format  (30  DRs  from  3  Users) 


Functional 

Area 

DRs 

Total 

SLOC 

Total 

RPN 

Communications 

7 

2200 

5240 

Navigation 

7 

1700 

4210 

Planning 

8 

4700 

3620 

Security 

5 

3550 

2720 

Propulsion 

3 

1450 

2100 

13600 


Highest  RPN  areas 
are  Communications 
and  Navigation 

Assuming  3000  SLOC 
per  build  you  could 
close  all  the  DRs  in 
Communications,  but 
you  will  need  to  do  a 
partial  fix  in  the 
Navigation  Area 
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Draft  Analysis  Method  -  Functionality 


DR# 

User  Priority 

Area 

SLOC 

RPN 

User  Top  3  Priority 

120 

A2 

Communications 

250 

1200 

114 

A3 

Communications 

1000 

800 

RPN  >1000 

116 

B5 

Communications 

200 

2000 

RPN  <500 

121 

A6 

Communications 

100 

200 

100 

A8 

Communications 

400 

160 

SLOC  >  500 

123 

B8 

Communications 

50 

400 

115 

C9 

Communications 

200 

480 

102 

B1 

Navigation 

500 

1500 

106 

B2 

Naviqation 

100 

600 

107 

B3 

Navigation 

250 

200 

▲ 

108 

B6 

Navigation 

100 

250 

1  3,000  SLOC 

122 

B7 

Navigation 

100 

500 

1  Cut-Off 

101 

B9 

Navigation 

400 

360 

117 

B10 

Navigation 

250 

800 

3900 


First  Build  -  4  of  9  Top  3  User  Rankings,  All  Comm  DRs,  First  2 
Navigation  DRs  ;  All  3  Users  have  at  least  1  DR  fixed 
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SLOC 


Second  Analysis  Method  -  System  Risk 


RPN  vs.  SLOC 

1400 


1200 


1000 


800 


600 


B6  ♦ 

400  - AS — - B9  ♦ 


A6  ♦ 

B3  CIO  «C4  A5 
200  A7  C9 


A1  B7 
B8  ♦ 


500 


B10  ♦ 


A2  ♦ 


% 


C3  ♦ 


1000 


1500 


We  would  look  at 


B5 


2000 


2500 


RPN 
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Top  10  RPN  DRs 


DR# 

User  Priority  Area 

SLOC 

RPN 

User  Top  3  Priority 

116 

102 

B5 

B1 

Communications 

Navigation 

200 

500 

2000 

1500 

RPN  >1000 

113 

C6 

Security 

900 

1500 

RPN  <500 

120 

A2 

Communications 

250 

1200 

103 

C3 

Propulsion 

400 

SLOC  >  500 

114 

A3 

Communications 

1000 

800 

▲ 

117 

B10 

Navigation 

250 

800 

125 

B4 

Security 

450 

800 

118 

C2 

Planning 

1100 

800 

106 

B2 

Navigation 

100 

600 

5150 

3,000  SLOC 

First  Build 

-  3  of  9  Top  3  Priority  DRs, 

4  of  5 

Cut-Off 

functions,  burns  down  ~40%  of  total  system  risk 
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SLOC 


Third  Analysis  Method  -  User  Ranking 


RPN  vs.  SLOC 

1400 


1200 


1000 


800 


A 

-A - ♦- 


600 


400 


♦  A  ■  " 

200 


♦  A 


500 


1000  1500 

RPN 


2000 


Concentrate  on  the  blue  diamonds  first 


♦  Priority  1-3 
■  Priority  4-6 
A  Priority  7-10 


2500 
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Top  User  Ranked  DRs 


DR# 

User  Priority 

Area 

SLOC 

RPN 

User  Top  3  Priority 

124 

A1 

Planning 

100 

400 

RPN  >1000 

102 

B1 

Navigation 

500 

1500 

RPN  <500 

127 

Cl 

Propulsion 

800 

600 

120 

A2 

Communications 

250 

1200 

SLOC  >  500 

106 

B2 

Navigation 

100 

600 

118 

C2 

Planning 

1100 

800 

114 

A3 

Communications 

1000 

800 

i 

107 

B3 

Navigation 

250 

200 

103 

C3 

Propulsion 

400 

1200 

3,000  SLOC 

Cut-Off 

First  Build  -  6  of  9  Top  3  Priority  DRs, 
4  of  5  functions 
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Hybrid  Method  -  Start  with  User  Ranking 


DR# 

User  Priority 

Area 

SLOC 

RPN 

User  Top  3  Priority 

124 

A1 

Planning 

100 

102 

B1 

Navigation 

500 

1500 

RPN  >1000 

127 

Cl 

Propulsion 

800 

600 

1200 

RPN  <500 

120 

A2 

Communications 

250 

106 

B2 

Navigation 

100 

600 

SLOC  >  500 

118 

C2 

Planning 

1100 

800 

114 

A3 

Communications 

1000 

800 

107 

B3 

Navigation 

250 

200 

103 

C3 

Propulsion 

400 

1200 

126 

A4 

Security 

400 

100 

125 

B4 

Security 

450 

800 

129 

C4 

Planning 

250 

400 

Based  solely  on  User  Rankings  you  would  fix  all  the  users’  top  2  DRs  -  BUT 
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Hybrid  Method  -  Then  Consider  Functionality 


Look  at  top  level  data  in  a  summary 
format  (30  DRs  from  3  Users) 


Functional 

Area 

DRs 

Total 

SLOC 

Total 

RPN 

Communications 

7 

2200 

5240 

Navigation 

7 

1700 

4210 

Planning 

8 

4700 

3620 

Security 

5 

3550 

2720 

Propulsion 

3 

1450 

2100 

13600 


Based  solely  on  User  Rankings  you  would  fix  all  the  users’  top  2  DRs  -  BUT 

There  are  only  3  Propulsion  DRs  total  and  2  were  top-3  priority  list  -  the  total 
SLOC  for  all  three  is  1450  so  you  might  consider  doing  those  first 
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Hybrid  Method  -  Determine  What  Else  To  Include 

Based  solely  on  User  Rankings  you  would  fix  all  the  users  top  2  DRs  -  BUT 


There  are  only  3  Propulsion  DRs  total  and  2  are  in  this  list  -  the  total 
SLOC  for  all  three  is  1450  so  you  might  consider  doing  those  first 


You  could  then  add  in  6  of  the  7  Navigation  DRs  and  still  be  under  the  3000 
SLOC  budget 
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Hybrid  Method  -  Final  Listing 


DR# 

User  Priority 

Area 

SLOC 

RPN 

127 

Cl 

Propulsion 

800 

600 

103 

C3 

Propulsion 

400 

1200 

112 

CIO 

Propulsion 

250 

300 

102 

B1 

Navigation 

500 

1500 

106 

B2 

Navigation 

100 

600 

107 

B3 

Navigation 

250 

200 

108 

B6 

Navigation 

100 

250 

122 

B7 

Navigation 

100 

500 

117 

B10 

Navigation 

250 

800 

User  Top  3  Priority 

RPN  >1000 

RPN  <500 
SLOC  >  500 


Based  solely  on  User  Rankings  you  would  fix  all  the  users  top  2  DRs  -  BUT 

There  are  only  3  Propulsion  DRs  total  and  2  are  in  this  list  -  the  total 
SLOC  for  all  three  is  1450  so  you  might  consider  doing  those  first 

You  could  then  add  in  6  Navigation  DRs  and  1300  SLOC  (2750  total  SLOC) 


Note:  You  could  add  additional  DRs  to  get  to  3000  SLOC;  or  you  could  have 
considered  adding  Communication  DRs  next  instead  of  Navigation 
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Other  uses 


Can  be  used  in  a  development  environment: 

•  Severity  can  be  related  to  test  blockers  or  number  of  interfaces  to 
other  units,  to  key  requirements  or  to  operational  impacts  (if 
known) 

•  Detection  still  based  on  ability  to  know  the  defect  has  occurred 

•  Time  can  be  based  on  the  effort  needed  to  correct  the  defect 

•  RPN  can  still  be  compared  to  functionality  and  to  total  cost  to  fix 

Can  be  used  in  a  maintenance  environments 

•  Rating  scale  development  would  be  very  similar  to  the  example 

•  Would  tend  to  try  to  fix  the  highest  RPN  defects  first,  but  may  still 
group  by  functionality  or  users  depending  on  the  situation 
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Suggestions  for  DoD  Usage 


Develop  a  team  to  put  together  the  structure  for  RPN  use 

•  Include  the  program  office,  using  command,  users,  contractors, 
etc.  as  needed 

Need  to  develop: 

•  Definitions  for  severity  which  may  include  different  categories 

•  Definitions  for  detection  which  may  include  different  categories 

•  Methods  for  dealing  with  occurrence  measures 

•  Scaling  factors 

•  Computation  methods 

•  Data  collection  methods 

•  Process  for  using  RPN  values 
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Questions? 
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Contact  Information 


Presenter  /  Point  of  Contact 

Julie  Cohen 

Client  Technical  Solutions  Division 
Telephone:  +1  412.268.3950 
Email:  icohen@sei.cmu.edu 

Customer  Relations 

Email:  info@sei.cmu.edu 
Telephone:  +1  412.268.5800 

SEI  Phone:  +1  412.268.5800 

SEI  Fax:  +1  412.268.6257 


William  Hayes 

Client  Technical  Solutions  Divisions 
Telephone:  +  1  412.268.6398 
Email:  wh@sei.cmu.edu 
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Additional  Rating  Scale  Examples 

Backup  Materials 
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Rating  Scales  -  Severity 


a-1  Minor  Data  Issue 

a-2  Missing  Important  or  Incorrect  Data 
recoverable  without  using  manual  changes 
of  data  products 

a-3  Missing  important  data  or  some  data 
incorrect  -recoverable  using  manual 
changes  of  data  products 

a-4  Missing  important  data  or  some  data 
incorrect  but  some  data  is  fine  -  non- 
recoverable 

a-5  Recoverable  Corruption  using  manual 
changes  of  data  products 

a-6  Unrecoverable  Corruption 

N/A 
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Rating  Scales  -  Severity 


Data 

Fault 


System 
Crash 


System 

Function 


b-1  Crash  -  automatic  restart 

b-2  Recoverable  Application  Crash  -  Simple 
Recovery 

b-3  Recoverable  Application  Crash  - 
Complex  Recovery 

b-4  Recoverable  System  Crash  -  Simple 
Recovery  Steps  - 

b-5  Recoverable  System  Crash  -  Complex 
Recovery  Steps 


b-6  Unrecoverable  System  Crash 


N/A 
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Rating  Scales  -  Severity 


Data 

Fault 


System 

Crash 


System 

Function 


c-1  Minor  System  Malfunction 

c-2  System  Malfunctions  or  Fails  to  Execute 
Some  Functions  but  work-around  exists 

c-3  Interruption  in  System  Functionality 
Requiring  operator  intervention 

c-4  Interruption  in  System  Functionality 
Requiring  contractor  Intervention 

c-5  Severely  Constrained  System 
Functionality  -  difficult  work-arounds  needed 

c-6  No  functionality  is  available  and  task 
cannot  be  performed  by  any  method 

N/A 
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Rating  Scales  -  Operational  Impact 


A 

( 

ups 

mpac 

¥ 

Operational 

Impact 
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Rating  Scales  -  Operational  Impact 


A 

ups 

npac 

W 

d-1  Increases  operator  workload  slightly 
d-2  Increases  operator  workload  significantly 
d-3  Could  limit/delay  mission  operations 
d-4  Certain  delay/limit  to  mission  operations 
d-5  Could  cause  mission  failure 
d-6  Certain  mission  failure 
N/A 
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Rating  Scales  -  Detection 


User 

Visibility 


I 


Data 

Issues 


Security 

Risk 


Workaround 

Risk 
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Rating  Scales  -  Detection 


User 

Visibility 


Data 

Issues 


I 


Security 

Risk 


Workaround 

Risk 


e-1  There  is  an  explicit  alert  or  warning  that 
there  is  a  malfunction.  Or  the  system  or 
application  fails  or  crashes. 

e-2  Users  will  always  notice  a  visible 
malfunction,  and  only  novices  would  fail  to 
detect  the  unexpected  system  behavior. 

e-3  Users  will  always  notice  a  visible 
malfunction,  but  only  after  other  functions  or 
workflow  steps  have  completed. 

e-4  A  user  may  detect  subtle  symptoms 
during  normal  operation,  but  may  not 
immediately  recognize  the  cause. 


e-5  Issue  not  detectable  during  normal 
operation 


N/A 
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Rating  Scales  -  Detection 


User 

Visibility 


Data 

Issues 


I 


Security 

Risk 


Workaround 

Risk 


f-1  The  system  provides  a  warning  or  alert 
that  data  corruption  has  occurred. 

f-2  There  is  data  corruption  which  is  revealed 
to  the  user  by  an  obvious  malfunction  or 
erroneous  system  output. 

f-3  There  is  data  corruption  visible  only  after 
a  system  function  or  workflow  step  have 
revealed  the  corruption. 

f-4  There  is  a  data  corruption  which  can  be 
detected  only  by  specialized  staff  (e.g., 
expert  user) 

f-5  There  is  data  corruption  that  remains 
undetectable  to  the  user. 


N/A 
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Rating  Scales  -  Detection 


User 

Visibility 


Data 

Issues 


I 


Security 

Risk 


Workaround 

Risk 


g-1  The  system  provides  a  warning  or  alert 
regarding  the  security  issue. 

g-2  There  is  a  visible  security  issue  which  is 
easily  detected  by  the  user. 

g-3  There  is  a  security  issue  which  can  be 
detected,  but  only  after  another  system 
function  or  workflow  step  has  completed. 

g-4  There  is  a  security  issue  which  can  be 
detected,  but  only  with  involvement  of 
specialized  staff  (e.g.,  expert  user), 

g-5  There  is  a  security  issue  which  is  not 
visible  to  the  user 


N/A 
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Rating  Scales  -  Detection 


h-1  The  work-around  impacts  large  areas  of  system 
function,  so  an  unsuccessful  work-around  has 
greater  impact 

h-2  The  work-around  requires  specialized  expertise 
to  accomplish  which  may  not  be  readily  available 
when  needed 

h-3  Work-around  implementation  blocks  all  other 
work  on  the  MPE  system  (for  example,  planning 
can’t  continue  while  a  crypto  work-around  is  being 
implemented) 

h-4  The  workaround  requires  changes  in  more  than 
one  part  of  the  workflow  to  be  accomplished  to 
ensure  the  work-around  is  effective 

h-5  Work-around  is  very  error  prone  and  there  is 
high  probably  that  the  work-around  will  be 
ineffective  or  will  cause  unanticipated  side-effects 
that  will  negatively  impact  operations 

N/A 
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